1. Field of the Invention
This invention pertains in general to computer security and in particular to detecting and responding to attempted exploits of vulnerabilities of applications and other programs executing on a computer.
2. Description of the Related Art
Applications executed on modern computers are often susceptible to a wide variety of network-based attacks. Web browsers, for example, are particularly susceptible to attacks because browsers receive large amounts of content from the Internet. The content can include hypertext markup language (HTML) web pages, scripts in languages such as VBScript and JavaScript, and executable content. Other types of applications are also vulnerable. For example, email programs and even word processors provide interfaces for executing network-based content.
Malicious attackers can compromise such applications by crafting specially-formulated inputs that exploit vulnerabilities in the programs. Each input contains code that, when executed, gives the attackers control over the applications and allows them to perform malicious acts such as capturing keystrokes, sending messages on the network, deleting files, installing malicious software (malware) such as spyware and adware, etc.
One technique for detecting and preventing these malicious exploits is scanning network traffic entering the computer in order to detect malicious code. For example, an intrusion detection system (IDS) can scan network traffic entering an enterprise network for characteristics of malicious code, and then prevent the code from entering the enterprise. A problem with this approach is that attackers use classical obfuscation and encryption techniques to evade detection. For example, an attacker can create a seemingly-innocent script that passes through the IDS but produces a malicious script when executed by an application.
Security software is often configured to detect exploits at the application level due to the difficulty with network-based detection described above. Unfortunately, application-level detection means that when the exploit is detected, the vulnerable application may have already begun to process the input that constitutes the exploit. As a result, it is difficult to block the exploit without terminating or otherwise adversely impacting the application.